Skip to content

Phase 2, theme 1: retention markers + prune subprocess + Tools run record, and run identity (RUN-01..07) - #30

Merged
guijacquemet merged 2 commits into
mainfrom
claude/phase2-qupath-retention
Oct 13, 2026
Merged

Phase 2, theme 1: retention markers + prune subprocess + Tools run record, and run identity (RUN-01..07)#30
guijacquemet merged 2 commits into
mainfrom
claude/phase2-qupath-retention

Conversation

@guijacquemet

@guijacquemet guijacquemet commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Theme 1 of the wrong-science wave for QuPath: retention markers + the prune subprocess + the run record through the Tools helpers (SPEC_WRONG_SCIENCE section 3) and run identity (section 4). The reference is the Napari work (LabConstrictor-napari #51 and #57). Number limits (theme 2) and async lookups + compatibility + conformance tests (theme 3) follow in two more PRs that are stacked on this one (see the end).

Fixes #14. Refs #16 (already closed: this PR adds the end-of-run half of it).

What changes

Retention: the host never deletes, it marks and asks Tools

  • New LcRetention (file LcRetention.groovy): .lc-owner.json is written first when a folder is made (results folder in newJobDir, run-record folder at the start of the run), .lc-done.json last and atomically (temp file, then rename). The formats are those of section 3.2 / 3.8 (and checked by Tools itself in the tests, see below).
  • Done status = labconstrictor_tools.runs.done_status (COMPLETE ok, CANCELED cancelled, FAILED failed, CRASHED cancelled after Cancel and failed otherwise). A worker that was killed after Cancel keeps its crash text in the run record (status CRASHED) and the done status cancelled; closing the window during a run is a cancellation too (Napari does the same).
  • After both done markers the host starts <app python> -m labconstrictor_tools prune in a daemon thread: it does not wait, ignores the exit code except to log a failure (prune ended with exit code N, prune could not be started) and a failure never fails or delays the run. If the subprocess cannot start the folders stay: there is no code in the script that deletes anything under results/ or runs/ (a test scans the sources for it).
  • The old LcRuns moved to the new file and no longer writes the record itself: the record is built by the Tools helpers runs.build_record + record_json in the app's own Python (LcRunRecord, a small python -c helper), host: "qupath" is added, and run_id / started_utc are passed. If the helper cannot run, the status line says so (the run record could not be written (...); its folders stay as they are and are never removed) and the folders get no done marker (never pruned).
  • Tools too old: LcCompat (a Groovy port of compat.check_schema) refuses a run whose app has Tools older than 0.3.0a1 or without retention_prune, with the Tools sentence (this app's labconstrictor-tools 0.2.0rc1 is older than the 0.3.0a1 this host needs: update it). PR 3 applies the same check when the registry is read.
  • README and docs/user-guide: the old text said "the host never deletes ... (retention belongs to Tools)" while nothing could ever clean results/ and runs/ (no marker, so prune could never remove them). Rewritten. Folders written by older versions have no marker and are never cleaned automatically (spec 3.3, "legacy folders ... never auto-deleted"; Tools has a follow-up item clean --legacy).

Run identity (RUN-01..07, issue #16)

  • New LcRun (LcRunIdentity.groovy), frozen when Run is pressed on the FX thread, before any worker call: run_id, app, tool, the ImageData object of every "Current image" input (+ its shape), the ROIs of the selected annotations (immutable objects, so by value), the plane and the Image areas. Export, the RegionOf mask, the request, the run record and the placement of results read it, nothing live. The chooser text is never identity.
  • Placement draws on that very ImageData only while a viewer still shows it with the shape it was exported with; otherwise nothing is drawn, the files stay in the owned results folder, and the status is the conformance sentence target_unavailable (gone) or target_changed_size (still shown, other shape), never swapped, with the run_id and the path; the code [target_unavailable] is logged. An image that is open in a second viewer gets the results even if the active viewer shows another image.
  • Every drawn annotation carries labconstrictor.run_id, .app and .tool of the run (metadata), the results window keeps the run id, the run record has run_id and started_utc. A tool or app switched while a run goes on never changes what the run records or draws and its end no longer clears the fields of the tool shown now (ClearAfterRun).
  • The "Current image: ..." entry of every image chooser now follows the open image (before, the label kept naming the first image while the export took whatever was open). With no image open the entry keeps its label and a run is refused ('blobs.tif' is no longer the image open in QuPath (no image is open): choose the image again); the same refusal protects the case where the chooser and the open image disagree at the click.

Structure

  • The script is now several files (parts.txt: the main script first, then LcCompat, LcRetention, LcRunIdentity), joined by the extension, by tests/run_gui_test.sh and by build.sh (which also writes target/LabConstrictorTools-all.groovy for the Script Editor: the main script alone does not run any more). The main script grows by about 30 lines, all new code is in the new files (each below 600 lines, all new functions typed). lint.yml lints the joined text (CodeNarc reads one file at a time and cannot resolve a class of another file) and fails when a part file reaches 600 lines.

Tests

New bodies (CI registers the new runs test app from tests/runs_app: delayed tools that wait for a gate file, a tool that ignores Cancel, one that dies):

  • gui_test_retention_body.groovy: HRET-01..06, RET-10, RET-11 read from conformance.json; owner/done markers checked field by field and judged eligible by Tools itself (prune --dry-run --json); the prune replaced by a recorder (saw the done files), by a failing and by a missing command; 60 runs with foreign and 60/120 old folders surviving; the real prune (newest 20 / 50 kept, legacy, foreign, unfinished, renamed-copy and symlinked folders untouched, by file content); a worker killed after Cancel (cancelled, crash text in the record) and one that dies (failed); a failed and a cooperatively cancelled run; the temporary export folder removed; the Tools-too-old refusals; an unbuildable record; a scan that no code deletes under results/runs.
  • gui_test_identity_body.groovy: RUN-01..04, 06, 07 read from conformance.json through the real dialog (a gate file is the barrier): image switched, closed, kept in a second viewer; selection changed during the export (a hook on the run thread) for both the crop and the mask; tool and app switched while running; the shape changed by reflection (nothing else can change an ImageData's server); sentences compared with the conformance templates, code logged.

Existing tests changed (spec supersedes them)

  • gui_test_parity_support_body.groovy: LcRuns.record(...) now needs the app's real interpreter (it calls the Tools helper): the three calls pass the registered interactions entry; app_version is read from the registry by Tools (it used to be the value the test passed).
  • gui_test_parity_host_body.groovy, gui_test_parity_form_body.groovy, gui_test_parity_pixel_body.groovy: the synthetic schemas they register now say tools_version and features (a run of an app with Tools older than 0.3.0a1 is refused).
  • gui_test_frame_body.groovy, gui_test_area_body.groovy: they count the folders of the results folder, which the real prune (started after every run) now keeps at 20 in the shared registry; both switch the prune to a no-op so the count means what it did (the real prune has its own body).
  • tests/run_gui_ci.sh: two new bodies, and the two duplicated lines of the needs table removed.

Nothing was skipped or loosened.

Sabotage checks

(see the comment on this PR: each break of production code in a copy made the named test fail)

Spec gaps and conservative choices

  • RUN-05 (Napari layer moved) does not apply to QuPath. RET-14 (undisplayable image result) does not apply: a result is never over a display limit here and every file already lives in the owned results folder (so RET-10's "move into an owned folder" is the normal case). RET-01..09, 12, 13 are Tools' own fixtures.
  • RUN-07 cannot happen through the GUI in QuPath (an ImageData's server is fixed); the check is implemented against the same ImageData and tested by replacing its server with reflection.
  • "Still open" means "shown in any viewer, hidden grid viewers included".
  • The record folder is made right after the results folder, i.e. after the export succeeded (a refusal or a failed export leaves no record, as before); the spec says "when the run starts".
  • When the Tools helper cannot run, no done marker is written (conservative: the folders are never pruned) and the status says so.
  • started_utc/finished_utc: finished_utc is the host's own (ends in Z, as before) rather than the helper's default.
  • The wording of the no longer the image open refusal and of the record problem are this host's own.
  • Fixed the chooser/open image mismatch even though only the end-of-run half was asked: the label could not otherwise be trusted as a name.

Not verified

  • Windows and macOS (the atomic rename of the done marker and the symlink cases are POSIX-tested only; Tools judges the markers).
  • No human at a display: everything ran under a private Xvfb with a real QuPath 0.7.0 (Java 25).
  • The old hs_err_pid*.log files are untouched here (PR Remove two committed JVM crash logs and ignore them #29 removes them).

Stack

PR 2 (number limits + exit-code rule) is based on this branch and PR 3 (async + compatibility + conformance) on PR 2. Merge this one as a merge commit and retarget the next.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN

Generated by Claude Code

@claude
Retention markers, the prune subprocess, the run record through the T…
01f2625 
…ools helpers, and run identity

Hosts never delete under results/ and runs/: the host writes .lc-owner.json when a folder is made and
.lc-done.json (atomically) when the run ends, then starts `labconstrictor-tools prune` in a subprocess.
The done status and the run record come from labconstrictor_tools.runs (Tools >= 0.3.0a1, checked).
A run is frozen when Run is pressed (ImageData objects, selected ROIs, plane, areas) and placed only on
that image while it is still shown with its exported shape (target_unavailable / target_changed_size).
The script is split into files listed in parts.txt; lint joins them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN
This was referenced Oct 13, 2026
Merged
Merged
@guijacquemet Claude

guijacquemet commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Sabotage checks

Each row: production code broken in a copy of the final tree of this PR (a regular expression replaced one place), the PR's own GUI body run on it. A row must show failed checks; every row does. "aborted with an exception" means the test body died on the missing marker or folder (the body's own read of it raised), which also fails the body (exit code 1, no RESULTS line of passes).

sabotage (production code broken in a copy) what the test run said
the results folder's owner marker says kind run 6 failed check(s): HRET-03_result_owner_marker_is_valid, HRET-03_tools_itself_finds_both_folders_eligible, HRET-02_the_markers_of_all_60_new_folders_are_valid_for_tools ...
the done marker always says ok 4 failed check(s): HRET-05_both_owned_folders_are_cancelled_not_failed, HRET-06_both_owned_folders_are_failed, RET-11_a_failed_run_is_failed ...
the host never starts prune 3 failed check(s): HRET-04_the_recorder_saw_prune_after_the_done_files, HRET-04_a_failing_prune_does_not_fail_the_run, HRET-04_a_prune_that_cannot_start_does_not_fail_the_run_and_is_said
prune is started before the done markers are written 2 failed check(s): HRET-04_the_recorder_saw_prune_after_the_done_files, HRET-04_each_prune_saw_the_runs_done_files
the host deletes the oldest results folder after each run 1 failed check(s): HRET-03_results_folder_is_named_like_the_command_line
the host deletes the oldest results folder once there are more than 70 test aborted with an exception (rc=1 after 19 checks; the seeded sentinel of a deleted folder could not be read)
cancelRequested is always false (Cancel is forgotten) 1 failed check(s): HRET-05_both_owned_folders_are_cancelled_not_failed
a worker killed after Cancel loses its crash text 1 failed check(s): HRET-05_the_crash_text_is_in_the_run_record
the run record of a killed-after-Cancel worker says CANCELED instead of CRASHED 1 failed check(s): HRET-05_the_crash_text_is_in_the_run_record
the run-time Tools check is switched off 4 failed check(s): compat_runs_old_is_refused_with_a_clear_sentence, compat_runs_nofeature_is_refused_with_a_clear_sentence, compat_runs_unknown_is_refused_with_a_clear_sentence ...
the results folder gets no owner marker test aborted with an exception (rc=1 after 5 checks)
the results folder gets no done marker test aborted with an exception (rc=1 after 5 checks)
targetProblem never reports a gone image 5 failed check(s): RUN-01_the_status_is_the_target_unavailable_sentence, RUN-01_nothing_is_drawn_on_A_either_it_is_not_open, RUN-04_the_code_is_logged_and_no_exception_was_raised ...
the target_unavailable and target_changed_size sentences are swapped 2 failed check(s): RUN-01_the_status_is_the_target_unavailable_sentence, RUN-04_closing_A_gives_the_target_unavailable_sentence_and_keeps_the_files
results are placed on the image that is open at the end, not the run's image 9 failed check(s): RUN-01_the_status_is_the_target_unavailable_sentence, RUN-01_nothing_is_drawn_on_B, RUN-04_the_code_is_logged_and_no_exception_was_raised ...
the selection is read live at export (not the click's) 1 failed check(s): RUN-02_the_mask_is_the_selection_of_the_click
the selected ROIs are not captured at the click 3 failed check(s): RUN-02_the_exported_crop_is_the_selection_of_the_click, RUN-02_the_results_are_placed_with_the_offset_of_the_click, RUN-02_the_mask_is_the_selection_of_the_click
the end of a run clears the form of whatever tool is shown 1 failed check(s): RUN-03_the_form_of_the_other_tool_is_not_cleared_by_the_end_of_the_run
annotations do not carry the run id 5 failed check(s): RUN-06_the_point_carries_the_run_id_the_app_and_the_tool, RUN-06_run_2_lands_on_B_with_its_own_run_id, RUN-01_A_still_open_in_another_viewer_gets_the_points_and_the_active_B_gets_nothing ...
a chooser label that names another image than the open one is not refused 1 failed check(s): a_stale_chooser_label_is_refused
the image chooser does not follow the open image 2 failed check(s): RUN-01_the_chooser_follows_the_open_image, RUN-06_run_2_lands_on_B_with_its_own_run_id
the run record has no run_id / started_utc 4 failed check(s): the_run_record_has_the_same_run_id, RUN-04_the_run_record_is_written_with_the_run_id, RUN-03_the_run_record_names_the_tool_of_the_run ...

(The two host_deletes rows: the second one deletes only once more than 70 results folders exist, to reach HRET-02; the body died when its seeded sentinel file was gone.) One more sabotage of the first kind was tried and did NOT come out the way I wanted: deleting the oldest folder after every run kills the very first assertion, so it shows nothing about HRET-01/02 specifically.

Generated by Claude Code

@claude
Header of the script says it is several files now; the retention body…
8ad2dc0 
… reports a missing file as a failed check

The header still said the script could be pasted into the Script Editor alone. The retention body read the
sentinel and marker files of folders with plain reads, so a deleted folder killed the body with an exception
instead of failing the named check (found by sabotage).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN
@guijacquemet
guijacquemet merged commit 3ea1658 into main Oct 13, 2026
2 checks passed
@guijacquemet
guijacquemet deleted the claude/phase2-qupath-retention branch October 14, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Protect unrelated and active result folders from QuPath retention cleanup

2 participants

@guijacquemet @claude